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Informatics 


comment on the design and validity of the conclusions of the 2017 Compliance 


of compliance 


Audit Brieflv the audit was mtenaea 10 DC »-- * 

with the Oregon Forest Practices Act (FPA). A sample of “forest operation sites” was selected 
from the Forest Activity Electronic Reporting and Notifications System (FERNS) and was use 

to assess compliance with FPA rules. 


A primary advantage of a properly designed and executed statistical survey is the ability to 
assess precision, e.g., by calculating confidence intervals. Unfortunately, the selection process 
used in the survey has severe flaws which were compounded by the analysis. These flaws are 
sufficiently severe to preclude any defendable assessment of compliance rates. The audit 
program should not proceed without the advice of a qualified statistician with experience in 

survey design. 


Even if the design had been correctly implemented, the analysis used is not amenable to 
calculating precision or confidence interval estimates. Compliance rate was calculated for a 
stratum by dividing total number of non-compliant rule applications by total number of rule 
applications. There is certainly covariance among rule applications on a site so that a legitimate 



confidence is difficult or impossible to calculate. The usual binomial based P 

not apply. 


Comments of Study Design 


st steps in selecting a proper statistical sarnie every unit in tl 
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FERNS 
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sample, and the methods need to be carried through 


The sample 


received during 


There is no 
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reporting element is based on acreage, fostea t erepo mg additional stratification by 

(number of non-compliant sites divided by total number <?f site '* ownership class 

ownership class was earned out. It is not stated whether the 

was a proper stratification or whether it was simply a sub.sample of the existing P 

Details of the stratification process are not provided, but it does not appear to have been done 
Details or me sirauue* _ the total number of units in 


stratum 
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stratum. This information is necessary to correctly analyze a stratified samp e 


variable weights in the sample 


variation is ignored 




the analysis. 

Every site in the initial draw should be cross-classified as in Table 1, i.e„ by area and landowner 
class. Once Ihis is done, ,hen a target sample size should be specified for each stratum and 

samples for each stratum should be drawn independently. 


The most serious problem with the study is the high rate of non-response. Of the 354 sites in the 
initial sample, 111 were classified as non-target and excluded; sampling permission was refused 
at 40 sites, and no response was received for 79 sites (for a total of 119 sites for which no 
response was received). Permission to sample was received for only 115 sites. This provides an 

enormous potential for non-response bias. 


When data are missing completely at random (MCAR), the reason for the lack of response is not 
associated with the outcome of interest, any related covariates, or the survey design. The data 
are considered a random subsample of the intended sample and analysis can be conducted using 
standard approaches on the reduced data set. When data are missing at random (MAR), the 
missingness mechanism is related to levels of one or more covariates. Within these levels, the 
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assumed 


1 • under the MCAR assumption that the non-response sites 

In this case, the analysis proceed assumption does not seem 
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survey are not credible. There are some techniques 
nonse rate. For example, extreme efforts to obtam 
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offering rewards 


The report notes that “When a site was found to be unsuitable, or landowner P—« „ 

not be obtained, replacement sites were chosen using the random process descrtb p J- 

he hL hv simnlv selecting more samples. The process must begin with the initial selection by 




samples that required for the final sample 


Comments o 


analysis 


Most importantly, the analysis of a stratified random sample MUST take into account the 
stratification. Stratification introduces differential weight between strata, and this leads to bias if 
not accommodated in the analysis. Normally, results are calculated within strata, and then 
combined to obtain population level results. In particular, it is wrong to simply pool all 
observations to calculate an overall compliance rate. Observations of multiple applications 

within a site are unlikely to be independent. 


A more basic concern is the definition of compliance rate itself. Compliance rates were 
calculated based on the total number of potential rule applications for a given stratification (e.g., 
by Area, ownership class). The number of non-compliant applications was then divided by this 
number. Aside from the need to do a stratified analysis, there are serious problems with this 
approach. A memo from Brenda McComb to Lena Tucker mentioned pseudo replication and 
confidence interval calculation. These are both legitimate concerns. Almost certainly, the rule 
applications on a site are not independent, and cannot be analyzed as if they were. In particular, 
the ratio of non-compliant to total rule applications cannot be treated as binomial to calculate 



confidence. In fact, obtaining a valid confidence interval for a rate calculated as in the report 
would be extremely complicated or impossible, even if the audit had been properly designed. 

Some serious consideration needs to be devoted to developing an appropriate metric for a 
composite compliance rate that can be applied in a stratified design. A possible approach would 
be to derive an aggregate measure of compliance at the site level, and then combine over sites to 
obtain a population-level metric. This approach was used in ODF 2002. With a site-level metric 
of compliance, then confidence intervals can be calculated using the standard stratified sample 
formulae. 

Reference 

ODF 2002. Oregon Department of Forestry, 2002. Best Management Practices Compliance 
Monitoring Project: Final Report. April 2002. 75 pp. 

http://www.oregon.gov/ODF/DocumentsAVorkingForests/BMPComplianceReport.pdf 



